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REMARKS 

Claims 1-25 are pending, all of which the Office Action identified as allowable. 

However, the Office Action indicated that the Specification fails to comply with 37 
C.F.R. § 1.52(b) because the paragraph numbering does not consist of at least four numbers 
enclosed in square brackets, including leading zeros. Applicants, therefore, have submitted a 
substitute Specification in accordance with 37 C.F.R. § 1.125 that includes proper paragraph 
numbering. Although not required. Applicants also have included a markup of the substitute 
Specification showing that only paragraph numbers have been changed. Accordingly, the 
substitute Specification does not add any new matter. 

The Office Action objected to claim 25 because it is directed to a computer program 
product that comprises a series of steps. In order to remove any potential ambiguity, the claim 
has been amended to recite A computer program product comprising a computer readable 
medium, on which a computer program is stored, the computer program causing a computer to 
execute the text mining method as recited in claim 1 . Applicants note for the record that the 
amendment to claim 25 merely clarifies the statutory classification of the claimed subject matter, 
which previously recited limitation identical to the limitation recited in claim 1, and therefore 
does not narrow the scope of the claim or otherwise evince an intent to surrender any subject 
matter. 

In the event that the Examiner finds any remaining impediment to a prompt allowance of 
this application that may be clarified through a telephone interview, the Examiner is requested to 
contact the undersigned attomey. 
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[0Q0114 O34This application claims priority to Japanese Patent Application 
No. 2000-197421 filed on June 29, 2000, the contents of which is incorporated herein 
by specific reference. 



f00021f 02tThe present invention relates to a text mining method and apparatus for 
extracting features of documents. In particular, the invention relates to a text mining 
method and apparatus for extracting features of documents, wherein features are 
extracted such that all mutually associated documents and terms are placed near each 
other in the feature space. Applications of the invention include document and/or web 
retrieval, associated term retrieval, document classification. 

2. DESCRIPTION OF THE RELATED ART 

fOQ03]f O^In text mining as a technology for squeezing desired knowledge or 
information by making analysis of text data, effective feature extraction of documents is 
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an important task for efficiently performing document and/or web retrieval, associated 
term retrieval, document classification and so on. As a typical document feature 
extracting method, a vector-space model as set out on page 313 of "Automatic Text 
Processing" (Addison- Wesley, 1989) is fi*equently used. 

f00O41f O4Hn the vector-space model, when terms selected as indices in the documents, 
namely index terms representing the contents of the documents, are t in number, a 
vector Vi is used respectively to correspond to an index term T\ to define a 
t-dimensional vector space. All vectors forming thus the defined vector space can be 
expressed as a linear combination of t in number of the vectors corresponding to t in 
number of the index terms. In this vector space, a document Dr is expressed as 
follows: 

D. = S X,, V, (1) 

fOOOSlf OStIn the foregoing expression (1), Xir active on Vi is the contribution of the 
index term T\ to the document Dr and represents a feature of the document. The 
feature is an amount representing the term frequency of the index term in the document. 
A vector [Xri, Xr2, ... Xn]* of t x 1 (t rows and one column) becomes a featxire vector of 
the document Dr. As the simplest case, when the index term Tj appears in the 
document Dr, Xir is set to 1 . When the index term Tj does not appear in the document 
Dr, Xir is set to 0. In a more complicated case, as set forth in the foregoing publication 
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on page 279 to 280, two quantities are used. These two quantities are a term frequency 
tfri of the index term Tj in the document Dr and a document frequency dfi of documents 
containing the index term Ti in all documents registered in the document database. 
[00061f 06^For the group of documents consisting of d in number of documents, a t x d 
term-document matrix X can be defined as follows: 

X = [Xi, X2, Xd] 

[00071{07}Here, a t-dimensional vector xj = [xji, Xj2,..., XjJ* expresses the feature vector 
of the document Dj, and *(dash) represents matrix inversion. 

i00081{08}Fig. 1 is an illustration showing one example of documents, translated from 
Japanese sentences, registered in a document database, where "ronin" is a romanized 
word meaning students who, having failed a school entrance-exam of a particular 
academic year, are preparing for one next year. Fig. 2 is an illustration showing one 
example of a term-document matrix taking the Kanji (Chinese) characters appearing on 
the documents shown in Fig. 1 as index terms. Kanji terms are imderlined in Fig. 1. 
In Fig. 2, among a character string "let me know about" appearing in all of the 
documents 1 to 3, the Kanji term "know" is checked off from the index terms. Fig. 3 is 
an illustration showing one example of an actual input question, translated from 
Japanese, from a user, where Kanji terms are underlined. If the index terms of Fig. 2 
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are used to express the question, the question can be expressed with the term-document 
matrix shown in Fig. 4. 

f00091f O9|In general, when the vector-space model is used, similarity sim (Dr, Dg) of 

two documents Dr and Dg can be expressed as follows: 

t 

Z ^Ir ^is 

Sim (D^, = (2) 

Jz S 

V I = 1 1 = 1 

[OOlOlf ftW When the similarity of the question and each document of Fig. 1 is 
judged on the basis of the meaning of the question of Fig. 3, the question of Fig. 3 is the 
most similar to the document 3 of Fig. 1 . However, using the feature vectors as shown 
in Figs. 2 and 4, the similarity of each document of Fig. 1 and the question of Fig. 3 is 
respectively sim(document 1, question) = 0.5477, sim(docimient 2, question) = 0.5477, 
sim(document 3, question) = 0.5477. hi short, all have the same similarity. 
[00111f Qj44 As a solution for such a problem, a method called Latent Semantic 
Analysis (LSA) was proposed in "Journal of the American Society for Information 
Science" 1990, Vol. 41, No. 6, pp. 391 to 407. This method extracts latent meaning of 
the dociunents on the basis of co-occurrences of the terms and is significantly 
outstanding in terms of retrieving efficiency. Here, "co-occurrences of terms" 
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represents a situation where the terms appear simultaneously in the same 
documents/statements . 

[0012H Oj4t The LSA extracts a latent semantic structure of the documents by 
performing singular value decomposition (SVD) for the term-document matrix. In the 
obtained feature space, mutually associated documents and terms are located near each 
other, hi 2i report placed in "Behavior Research Methods Instruments & Computers" 
(1991), Vol. 23, No. 2, pp. 229 to 236, retrieval using the LSA indicates a result of 30% 
higher efficiency in comparison with the vector-space model. LSA will be explained 
hereinafter in more detail. 

[00131f Oi^ In LSA, at first, singular value decomposition is performed for the t x d 
term-document matrix X as set out below. 

X = ToSoDo' (3) 
[0014H Oj4| Here, To represents an orthogonal matrix of t x m, So represents a square 
diagonal matrix of m x m with taking m in number of the singular values as the 
diagonal elements and setting 0 to the other elements. Do' represents an orthogonal 
matrix of m X d. In addition, let us assume that 0 > d > t, and arrange the orthogonal 
elements of So in descending order. 
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[OOISIW ^ Furthermore, in LSA, with respect to the feature vector Xq of t x 1 of a 
document Dq, the following conversion is performed to derive a LSA feature vector 



of n X 1 : 



(4) 



[00161^ 01^ Here, S is a square diagonal matrix of n x n taking the first to (n)th of the 
diagonal elements of So, and T is a matrix of t x n drawing the frrst to (n)th columns of 



To. 



[00171t Ol^ As an example, results of singular value decomposition of the 
term-document matrix shown in Fig. 2 are given below. The matrices Tq, So and Do 



are expressed as follows: 

0.1787 -0.3162 0.3393 

0.1787 -0.3162 0.3393 

0.1787 -0.3162 0.3393 

0.4314 -0.3162 -0.1405 

0.4314 -0.3162 -0.1405 

T„ = 0.1787 0.3162 0.3393 

0.1787 0.3162 0.3393 

0.4314 0.3162 -0.1405 

0.4314 0.3162 -0.1405 

0.1787 0.3162 0.3393 

0.2527 0.0000 0.4798 



So = 



2.7979 0 0 
0 2.2361 0 
0 0 1.4736 
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Do = 



0.5000 -0.7071 0.5000' 
0.5000 0.7071 0.5000 
0,7071 0.0000 -0.7071 



[OQlSlf M^ Let us assume that the dimension t of the LSA feature vectors is 2 and 
applying the foregoing expression (4) to each feature vector of the term-document 
matrix in Fig. 2. Then, the LSA feature vectors of the documents 1, 2 and 3 are 
respectively [0.5000, -0.7071]\ [0.5000, 0.7071]' and [0.7071, 0.0000]\ In addition, 
applying the foregoing expression (4) to the feature vector of Fig. 4, the LSA feature 
vector of the question from the user becomes [0.6542, 0]'. 

f00191f O3^ Applying the foregoing expression (2) to the LSA feature vectors 
obtained as set forth above, the similarity of the question of Fig. 3 and each document 
of Fig. 1, become respectively, sim(document 1, question) = 0.5774, sim(document 2, 
question) = 0.5774, and sim(document 3, question) = 1.0000. Thus, a result that the 
document 3 has the highest similarity to the question can be obtained. Considering a 
help system application or the like utilizing computer networks, an answer statement of 
the document 3 registered in the document database will be returned to the user who 
asked the question of Fig. 3. 

100201{020} For singular value decomposition, an algorithm proposed in "Matrix 
Computations", The Johns Hopkins University Press, 1996, pp. 455 to 457, is frequently 
used. In the report of "Journal of the American Society for Information Science" set 
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forth above, there is a statement that the value of the number of rows (or columns) n of 
the square matrix S is preferably about 50 to 150. In addition, in the foregoing report 
of "Behavior Research Methods, histruments, & Computers", it has been indicated that 
better efficiency can be attained by pre-processing using the term fi-equency or 
document frequency instead of simply setting each element of the feature vector to 0 or 
1 before performing LSA. 

[00211f O234 However, in the algorithm for singular value decomposition proposed in 
the foregoing "Matrix Computations", memory space in the order of the square of the 
number of index terms t (t^) is required at the minimum. This is because a matrix of t 
X t is utilized for bidiagonalization of a matrix in the process of calculation of basis 
vectors spanning a feature space from a given term-document matrix. The prior art is 
therefore not applicable to document database holding a very large number of terms and 
data. Furthermore, the prior art requires complicated operations of matrices 
irrespective of the number of data. 
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SUMMARY OF THE INVENTION 
[00221f O22t The present invention has been worked out in view of the problems set 
forth above. In a first aspect of the present invention, a text mining method is 
provided for extracting features of documents using a term-document matrix consisting 
of vectors corresponding to index terms representing the contents of the documents. In 
the term-document matrix, contributions of the index terms to each document act on 
respective elements of the term-document matrix. The method comprises: 

a basis vector calculating step of calculating a basis vector spanning a 
feature space, in which mutually associated documents and terms are located in 
proximity with each other, based on a steepest descent method minimizing a 
cost; 

a feature extracting step of calculating a parameter for normalizing 
features using the term-document matrix and the basis vector, and extracting the 
features on the basis of the parameter; and 

a term-document matrix updating step of updating the term-docimient 
matrix to a difference between the term-document matrix, to which the basis 
vector is not applied, and the term-document matrix, to which the basis vector is 
applied. 
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f00231f O2^ In a second aspect of the present invention, a text mining method is 
provided for extracting features of documents wherein the cost is defined as a 
second-order cost of the difference between the term-document matrix, to which the 
basis vector is not appUed, and the term-document matrix, to which the basis vector is 
appHed. 

[002414 0341 In a third aspect of the present invention, a text mining method is 
provided for extracting features of documents wherein the basis vector calculating step 
comprises: 

an initializing step of initializing a value of the basis vector; 

a basis vector updating step of updating the value of the basis vector; 

a variation degree calculating step of calculating a variation degree of 
the value of the basis vector; 

a judging step of making a judgment whether a repetition process is to 
be terminated or not using the variation of the basis vector; and 

a counting step of counting the nimiber of times of the repetition 
process. 

[00251f OaSt In a fourth aspect of the present invention, a text mining method is 
provided for extracting features of documents wherein the basis vector updating step 
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updates the basis vector using a current value of the basis vector, the term-document 
matrix and an updating ratio controUing the updating degree of the basis vector. 
f00261f 036t In a fifth aspect of the present invention, a text mining method is 
provided for extracting features of documents wherein when all basis vectors and 
normalizing parameters required in extracting the features have been already obtained, 
the calculation of the normalizing parameters in the basis vector calculating step and the 
execution of the feature extracting step are omitted. In addition, the feature extracting 
step extracts the features using the basis vectors and the normalizing parameters that 
have been already obtained. 

[00271f 02?t In a sixth aspect of the present invention, a text mining apparatus is 
provided for extracting features of documents using a term-document matrix consisting 
of vectors corresponding to index terms representing the contents of the documents. In 
the term-document matrix, contributions of the index terms to each document act on 
respective elements of the term-document matrix. The apparatus comprises: 

basis vector calculating means for calculating a basis vector spanning a 
feature space, in which mutually associated documents and terms are located in 
proximity with each other, based on a steepest descent method minimizing a 
cost; 
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feature extracting means for calculating a parameter for normalizing 
features using the term-document matrix and the basis vector, and extracting the 
features on the basis of the parameter; and 

term-document matrix updating means for updating the term-document 
matrix to a difference between the term-document matrix, to which the basis 
vector is not applied, and the term-document matrix, to which the basis vector is 
applied. 

[00281f 028t In a seventh aspect of the present invention, a text mining apparatus is 
provided for extracting features wherein the cost is defined as a second-order cost of the 
difference between the term-document matrix, to which the basis vector is not applied, 
and the term-document matrix, to which the basis vector is applied. 
[0Q291f O39| In an eighth aspect of the present invention, a text mining apparatus is 
provided for extracting features of documents wherein the basis vector calculating 
means comprises: 

initializing means for initializing a value of the basis vector; 

basis vector updating means for updating the value of the basis vector; 

variation degree calculating means for calculating a variation degree of 
the value of the basis vector; 
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judging means for making a judgment whether a repetition process is to 
be terminated or not using the variation of the basis vector; and 

counting means for counting the number of times of the repetition 
process. 

1003011©^©} In a ninth aspect of the present invention, a text mining apparatus is 
provided for extracting features of documents wherein the basis vector updating means 
updates the basis vector using a current value of the basis vector, the term-document 
matrix and an updating ratio controUing the updating degree of the basis vector. 
[003114 0^ hi a tenth aspect of the present invention, a text mining apparatus is 
provided for extracting features of documents wherein when all basis vectors and 
normalizing parameters required in extracting the features have been already obtained, 
the calculation of the normalizing parameters in the basis vector calculating means and 
the execution of the feature extracting means are omitted. Li addition, the feature 
extracting means extracts the features using the basis vectors and the normalizing 
parameters that have been already obtained. 

100321{©33} Li an eleventh aspect of the present invention, a computer program 
product is provided for being executed in a text mining apparatus for extracting features 
of documents using a term-document matrix consisting of vectors corresponding to 
index terms representing the contents of the documents. In the term-document matrix, 
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contributions of the index terms act on respective elements of the term-document 

matrix. The computer program product comprises: 

a basis vector calculating step of calculating a basis vector spanning a 
feature space, in which mutually associated documents and terms are located in 
proximity with each other, based on a steepest descent method minimizing a 
cost; 

a feature extracting step of calculating a parameter for normalizing 
features using the term-document matrix and the basis vector, and extracting the 
features on the basis of the parameter; and 

a term-document matrix updating step of updating the term-document 
matrix to a difference between the term-document matrix, to which the basis 
vector is not applied, and the term-document matrix, to which the basis vector is 
applied. 

[00331f O3^ The feature extracting apparatus disclosed in this specification is 
constructed by defining a cost as a second-order function of a difference between the 
term-document matrix, to which the basis vector is not applied, and the term-document 
matrix, to which the basis vector is applied. The apparatus merely requires the 
following means: 
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a. basis vector calculating means for calculating a basis vector 
applying a steepest descent method to the cost; 

b. feature extracting means for calculating a parameter for 
normalizing features using the term-document matrix and the basis vector, and 
extracting the features on the basis of the parameter; 

c. term-document matrix updating means for updating the 
term-document matrix to the difference between the term-document matrix, to 
which the basis vector is not applied, and the term-docimient matrix, to which 
the basis vector is applied, in order to prevent redundant extraction of features; 
and 

d. featxire extraction control means for controlling execution of 
respective means. 

[00341t 034t The basis vector calculating means repeats calculation on the basis of the 
input term-document matrix to finally derive one basis vector. The repetition process 



I ^ g 3 " is terminated when the variation degree of the basis vector becomes less than or equal to 
S < w 2 § H ^ predetermined reference value. 



[0Q351f O^ The feature extracting means calculates a parameter for normalizing the 
features on the basis of the input basis vector and the term-docimient matrix, and 
extracts a feature for each document. 
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f00361f O3^ The term-document matrix updating means updates the term-document 
matrix on the basis of the input basis vector. 

f0037H Oyyt The feature extraction control means repeats execution of each means 
until the number of the features defined by the user is satisfied. When the basis 
vectors and the normalizing parameters have been already calculated, execution of the 
basis vector calculating means and calculation of the normalizing parameters in the 
feature extracting means are omitted. Then, the feature extraction can be performed 
with the construction incorporating the already obtained basis vectors and the 
normalizing parameters. 

100381f038} According to the present invention, a text mining method for extracting 
features of documents using a term-document matrix consisting of vectors 
corresponding to index terms representing the contents of the documents, wherein 
contributions of the index terms act on respective elements of the term-document 
matrix, comprises the following steps: 

i. a basis vector calculating step of calculating a basis vector 
spanning a feature space, wherein mutually associated documents and terms are 
located in proximity with each other based on a steepest descent method 
minimizing a cost; 
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ii. a feature extracting step of calculating a parameter for 
normalizing features using the term-document matrix and the basis vector, and 
extracting the features on the basis of the parameter; 

iii. a term-document matrix updating step of updating the 
term-document matrix to a difference between the term-document matrix, to 
which the basis vector is not applied, and the term-document matrix, to which 
the basis vector is applied; and 

iv. a feature extraction control step of controlling execution of 
respective steps. 

[Q0391f O^ Therefore, conceming feature extraction of documents in text mining, the 
features having the same nature as those obtained by LSA can be extracted with smaller 
memory space than the apparatus or method executing LSA. In addition, specific 
software or hardware for extracting the features can be easily implemented. 
100401{040} The above and other objects, characteristics and advantages of different 
embodiments of the present invention will become more apparent from the following 
descriptions of embodiments thereof taken in conjunction with the accompanying 
drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 
[0041H 0434 Fig. 1 is an illustration showing one example of documents registered in 
a document database; 

[0Q421f O42t Fig. 2 is an illustration showing one example of a term-document matrix 
with taking Kanji terms appearing in the documents shown in Fig. 1 as index terms; 
f0Q431f O4^ Fig. 3 is an illustration showing one example of a question actually input 
by a user; 

[0Q441f O44t Fig. 4 is an illustration showing a term-document matrix obtained from 
the question in Fig. 3; 

f0045] f04^ Fig. 5 is an illustration showing one embodiment of a feature extracting 
apparatus according to the present invention; 

[0Q461f O46t Fig. 6 is an illustration showing one example of a hardware construction 
for implementing the present invention; 

[0047] f047t Fig. 7 is an illustration showing a structure of a term-document matrix 
data file; 

J[00481f048} Fig. 8 is an illustration showing a structure of a basis vector data file, 
wherein the calculated basis vectors are stored; 

100491{049} Fig. 9 is an illustration showing a structure of a feature data file; 
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[OOSOIffi ^ Fig. 10 is an illustration showing a structure of a normalizing parameter 
data file; 

fOOSllf OSH Fig. 11 is a flowchart showing calculation of a basis vector in basis 
vector calculating means; and 

[00521f 052t Fig. 12 is an illustration showing one example of an automatic document 
classifying system employing one embodiment of the feature extracting apparatus 
according to the present invention. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 
f00531f OS^ Fig. 5 is an illustration showing one embodiment of a feature extracting 
apparatus according to the present invention. As shown in Fig. 5, feature extraction 
control means 200 has term-document matrix updating means 210, basis vector 
calculating means 220, feature extracting means 230. The reference numeral 100 
denotes a term-document matrix data file, 300 denotes a basis vector data file, 400 
denotes a feature data file, 450 denotes a normalizing parameter data file. In the 
term-document matrix data file 100, a term-document matrix of collected document 
data is stored. The term-document matrix updating means 210 reads the 
term-document matrix from the term-document matrix data file 100, and transfers the 
read term-document matrix to the basis vector calculating means 220 and the feature 
extracting means 230 without updating the term-document matrix, in a first iteration 
process. 

[00541f 054t In a second and subsequent iteration processes, the terms-document 
matrix is updated on the basis of the basis vector transferred fi'om the basis vector 
calculating means 220. The result of updating is transferred to the basis vector 
calculating means 220 and the feature extracting means 230. The basis vector 
calculating means 220 calculates one basis vector through a repetition process based on 
the term-docimient matrix transferred fi"om the term-document matrix updating means 
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210. Then, the degree of variation of the basis vector in respective repetition is 
monitored for terminating the repetition process when the degree of variation becomes 
less than or equal to a predetermined reference value. The basis vector calculating 
means 220 stores the calculated basis vector in the basis vector data file 300 and in 
conjunction therewith, transfers the calculated basis vector to the term-document matrix 
updating means 210 and the feature extraction means 230. The feature extracting 
means 230 extracts one feature of each document on the basis of the term-document 
matrix transferred from the term-document matrix updating means 210 and the basis 
vector transferred from the basis vector calculating means 220. The result is stored in 
the feature data file 400, and also the parameter for normalizing the features is stored in 
the normalizing parameter data file 450. 

[0055H OS^ Execution of the term-document matrix updating means 210, the basis 
vector calculating means 220 and the feature extracting means 230 is taken as one 
iteration process. Number of times of iteration processes will be indicated by suffix i, 
and number of features designated by the user is indicated by suffix n. The feature 
extraction control means 200 repeats the process until a condition, i = n is satisfied. 
On the other hand, in a case where all of the required basis vectors and the required 
normalizing parameters have akeady been obtained, execution of the basis vector 
calculation means 220 and calculation of the normalizing parameters in the feature 
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extracting means 230 may be omitted. Therefore, in such a case, the feature extraction 
control means 200 may be constructed with the term-document matrix updating means 
210 incorporating the known basis vectors and normaHzing parameters, and with the 
feature extracting means 230. 

[00561f OS6} Fig. 6 is an illustration showing one example of a hardware construction 
for implementing the present invention. As shown in Fig. 6, the feature extracting 
apparatus includes the following components: 

a central processing unit (CPU) 10 performing control for the overall 
apparatus, 

a memory 20 for storing the program and providing a temporary data 
storage region required for executing the program, 

a keyboard 30 for inputting data, and 

a display 40 for generating a display screen. 
f00571 fO5?l The programs to be executed by the feature extraction control means 
200, the term-document matrix data file 100, the basis vector data file 300, the feature 
data file 400, and the normalizing parameter data file 450 are stored in the memory 20. 
f00581f 058t By taking this construction, the feature extraction is performed by CPU 
10 receiving the command fi*om the user through the keyboard 30, a mouse pointing a 
desired position on the display 40, or the like. It should be noted that, in the example 
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shown in Fig. 5, the feature extraction control means 200 has a stand-alone 
construction. However, the feature extraction control means 200 may be buih-in other 
systems. 

100591fO59} Fig. 7 is an illustration showing a structure of the term-document matrix 
data file. In Fig. 7, the reference numerals 101-1, 101-2, 101-d correspond to 
t-dimensional term-document data A consisting of d in number of data. Here, X = [xi, 
X2, ... Xd], Xj = [xji, Xj2, ... Xjt]' are defined to express the term-dociunent data A with a t x 
d matrix X. 

i00601f060} Fig. 8 is an illustration showing a structure of the basis vector data file 
storing the calculated basis vectors. In Fig. 8, the reference numerals 301-1, 301-2, ... 
301 -n correspond to t-dimensional basis vector data B consisting of n in number of data. 
The (i)th element 301-i corresponds to an output value of the basis vector calculating 
means 220 in the (i)th iteration process in Fig. 5. In the following disclosure, this 
element is expressed by a t x 1 column vector w\ = [wn, Wi2, Wit]'. 
f00611t O634 Fig. 9 is an illustration showing a structure of the feature data file. In 
Fig. 9, the reference numerals 401-1, 401-2, 401-n correspond to d-dimensional 
feature data C consisting of n in number of data. The (i)th element 401-i corresponds 
to an output value of the feature by the feature extraction means 230 in the (i)th iteration 



-23- 



DocketNo. 14321.31 



process in Fig. 5. This element is expressed by an 1 x d row vector yi = [yji, ya, 
Yid]. 

[00621f 06^f Fig. 10 is an illustration showing a structure of the normalizing 
parameter data file. In Fig. 10, the reference numerals 451-1, 451-2, 451-n 
correspond to normalizing parameter data D consisting of n in number of data. The 
(i)th element 451-i corresponds to an output value of the normalizing parameter by the 
feature extracting means 230 in the (i)th iteration process in Fig. 5. 
[00631 106^ Using the foregoing definitions, an implementation of feature extraction 
in the shown embodiment will be explained. The term-document matrix updating 
means 210 reads out X from the term-document matrix data file 100 only when i = 1, 
namely in the first iteration process, to store in a t x d matrix E without performing any 
arithmetic operation. Accordingly, E = [ei, 62, Cd], Cj = [eji, ej2, ejj = [xji, Xj2, 
Xjt]*. In order to prevent redundant extraction of the features extracted in the preceding 
iteration processes, E is updated in the (i)th iteration using the current value and the 
basis vector calculated in the immediately preceding iteration process. The result of 
updating is transferred to the basis vector calculating means 220. A value of E in the 
(i)th iteration, E(i), will be expressed by the following expression (5): 




X, for i = 1 

E0*-1) - w..,(w'._, E(/-l)), otherwise 



(5) 
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10064H064} Here, E(i) = [ei(i), e2(i), ed(i)], each element ej(i) of E(i) is defined by 
ej(i) = [eji(i), ej2(i), ^t(i)]*- Namely, when i > 2, the term-docximent matrix is 
updated to a difference derived by subtracting the term-document matrix, to which the 
basis vector is applied, from the term-document matrix, to which the basis vector is not 
applied. 

[0065H O6St Fig. 1 1 is a flowchart showing calculation of the basis vector in the basis 
vector calculating means. In Fig. 1 1, a value of Wj in the (k)th repetition is expressed 
by Wi(k) = [wii(k), Wi2(k), Wit(k)]'. At first, at step S500, the suffix k is initialized to 
1. Subsequently, the process is advanced to step S510 to initiahze respective element 
of Wi(l) with an arbitrary value between -C to C. Here, the value of C may be a 
positive small value, such as C = 0.01. At step S520, in order to calculate the basis 
vector spanning a feature space where mutually associated documents and terms are 
located in proximity with each other, a second-order cost expressed by the following 
expression (6) is provided. 



[00661 ffi66^^ Here, "terms are placed in proximity" means that the positions of the 
terms are close with each other within a feature space, and "documents are placed in 
proximity" means that the positions of terms included in respective documents are close 
in the feature space. On the other hand, a cost means an object to be minimized. In 




(6) 
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the shown embodiment, the cost is defined as a second-order function of the difference 
between the term-document matrix, to which the basis vector is not appHed, and the 
term-document matrix, to which the basis vector is applied, as expressed by the 
expression (6). Here, 

y im 

is the (m)th element of a 1 x d vector y- which is defined as follows: 

y,- =[yn,yi2. .yid] = >v,'E(0 (7) 

[00671f 067t For the cost, the steepest descent method is applied to update the value of 
Wi as expressed by the following expression (8). 

w,(fc-f 1) = yi,{k) +^(E(0 - yi,{k)^,{k)) z,{ky (8) 

a 

[00681f O68l Here, ii i(k) is an update ratio controlling the degree of updating in the 
(k)th repetition, which is initialized by a positive small value when k is 1 , such as ii 
j(l) = 0.1. Every time of increment of k, the value is decreased gradually, hi the 
alternative, it is also possible to set the value at a constant value irrespective of k. On 
the other hand, Zi(k) is defined as follows: 

Zi(k) = >Vi(k)'E(i) (9) 
f00691f 069t At step S530, 6 i(k) indicating the degree of variation of Wj is derived as 
follows: 
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f00701 f070t At step S540, a judgment is made whether the process is to be terminated 
or not on the basis of the value of 8 i(k). As a result of the judgment, if termination is 
determined, the process is advanced to step S560, and otherwise, the process is 
advanced to step S550. Here, in Fig. 11, j3 i is a positive small value, such as jS i = 1 
X 10-^ 

[OOTll WW At step S550, the value of the counter k is incremented by 1. Then, the 
process is retumed to step S520. At step S560, Wj is stored as the (i)th data of the basis 
vector data file 300. At the same time, Wj is transferred to the term-document matrix 
updating means 210 and the feature extracting means 230. hi the feature extracting 
means 230, the feature yi and the normalizing parameter p{ are calculated in the 
following manner. 

y;=y:fPi (11) 
[00721f O?at Here, pi is defined as follows: 




f00731 f0?3t The feature y; and the normalizing parameter pi are stored respectively in 
the feature data file 400 and the normalizing parameter data file 450 as the (i)th data. 
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[00741f 0?4t Fig. 12 is an illustration showing one example of an automatic document 
classifying system employing the shown embodiment of the feature extracting 
apparatus. In Fig. 12, the reference numeral 601 denotes term-document matrix 
calculating means, 602 denotes classifying means. The classifying means 602 may be 
implemented by a method disclosed in "Joumal of Intelligent and Fuzzy Systems", 
published on 1993, Vol. 1, No. 1, Pages 1 to 25. 

f00751 fO?5t The document data stored in the document database E is taken in the 
automatic document classifying system 600. In the automatic document classifying 
system 600, a term-document matrix is derived in the term-document matrix calculating 
means 601 . The result of calculation of the term-document matrix is transferred to the 
feature extraction control means 200. The feature extraction control means 200 
extracts the features from the received term-document matrix. The extracted result is 
output to the classifying means 602. In the classifying means 602, the resuh of 
classification is output on the basis of the input features. 

[00761f O?^ To evaluate the present invention, feature extraction of actual document 
data related to an entrance examination system was performed. It has been confimied 
that the present invention could extract the features of the same nature as those 
extracted using the conventional LSA. 
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100771{077} Next, concerning the size of the memory space, in a typically practical 
case where the number of terms t is significantly greater than number of documents d (t 
» d), the conventional LSA requires in the order of t^ of the memory size, the present 
invention merely requires the memory size in the order of t d for calculating respective 
basis vectors. Furthermore, in order to realize the prior art, a complicated matrix 
operation apparatus is required. The system according to the invention, however, can 
be easily realized with an apparatus that performs simple arithmetic operations. 
Namely, according to the present invention, the LSA feature extraction can be 
performed using a smaller memory space and a simpler program. In addition, this 
simple program may be loaded in a digital signal processor (DSP). Therefore, a 
specific chip for feature extraction can be produced easily. 

[00781f O7W Hereinafter, the results of respective means executing the shown 
embodiment of the feature extracting apparatus for the documents of Fig. 1 and the 
question of Fig. 3 will be shown. 

3g E 

S ^ ^ 1 3 ~ f00791 Wm A. Documents of Fig. 1 

w o w H 5 

U g < H -r- |3 

§ ^ > o § > 100801f080} First, let X denote the term-document matrix of Fig. 2. 

>; 2 5 J S G 
2 2 K a H S 
2 ^ 2 ^ ^ S 

1 i ^ i i ^ fOOSllW W I. First Iteration in Feature Extraction Control Means 200 (i = 1) 

2 - J 

o ^ < 
% < « 
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100821{082} According to the foregoing expression (5), the term-document matrix 



updating means 210 outputs E(l) expressed by the following expression to the basis 
vector calculating means 220 and the feature extracting means 230. 



E(l) = 



1 


0 
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0 
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0 


1 


0 
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1 
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0 


1 
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0 


1 


1 
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1 


1 


0 


1 


0 


0 


0 


1 



f00831f O83t In the basis vector calculating means 220, initialization is performed with 
setting the basis vector wi(l) at [0.0100, -0.0100, 0.0100, -0.0100, 0.0100, -0.0100, 
0.0100, -0.0100, 0.0100, -0.0100, 0.0100]', M 1 at a fixed value 0.1, j? i at 1 x 10"^ 
The calculation shown in Fig. 11 is performed for a himdred thirty-two times. Then, 
the basis vector w, = [0.1787, 0.1787, 0.1787, 0.4314, 0.4314, 0.1787, 0.1787, 0.4314, 
0.4314, 0.1787, 0.2527]'is stored in the basis vector data file 300, and output to the 
feature extracting means 230 and the term-document matrix updating means 210. 
f00841ffi 84t * First Repetition in Basis Vector Calculating Means 220 (k = 1) 
J00851I085} From the foregoing expression (8), 



J CO < H - 

5 ^ w o b 

^ 5 -J " u 

^ < 
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w,(2) = [0.0103 -0.0097, 0.0103, 0.0093, 0.0107, -0.0103, 0.0097, -0.0100, 0.0100, 
-0.0103,0.0103]' 

w,(2) - w,(l) = 10-^ X [0.3332, 0.3334, 0.3332, 0.6668, 0.6666, -0.3332, -0.3334, 
0.0001, -0.0001, -0.3332, 0.3332]' 
6 i(l) = 0.0103 

[008611 086} * Second Repetition in Basis Vector Calculating Means 220 (k = 2) 
100871{087} From the foregoing expression (8), 

w,(3) = [0.0107, -0.0093, 0.0107, -0.0085, 0.0115, -0.0107, 0.0093, -0.0100, 0.0100, 
-0.0107, 0.0107]' 

wi(3) - wi(2) = IQ-^ X [0.4110, 0.4112, 0.4110, 0.8001, 0.7998, -0.3665, -0.3668, 
0.0224, 0.0221, -0.3665, 0.3887]' 

6 i(2) = 0.0015 
f00881f O88t ~ syncopated ~ 

100891{089} * A Hundred Thirty-Second Repetition in Basis Vector Calculating 
Means 220 (k= 132) 

[009011 ^0} From the foregoing expression (8), 

Wi(133) = [0.1787, 0.1787, 0.1787, 0.4314, 0.4314, 0.1787, 0.1787, 0.4314, 0.4314, 
0.1787, 0.2527]' 
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wi(133) - w,(132) = 10-^ X [-0.3020, -0.3020, -0.3020, -0.3020, -0.3020, 0.3020, 
0.3020, 0.3020, 0.3020, 0.3020, 0.0000]' 

5 i(132) = 9.5500 x 10"^ 
[0091 If OW In the feature extracting means 230, the operations shown in the 
expressions (11) and (12) are performed for outputting: 

y, =[0.5000, 0.5000, 0.7071] 

and 

pi = 2.7979 

to the feature data file 400 and the normalizing parameter data file 450. 
100921f092} II. Second Iteration in Feature Extraction Control Means 200 (i = 2) 
[00931f O9^ In the term-document matrix updating means 210, fi"om the foregoing 
expression (5), E(2) expressed as follows is output to the basis vector calculating means 



-0.3536 
-0.3536 
-0.3536 
0.1464 
0.1464 
-0.3535 
-0.3535 
0.1465 
0.1465 
-0.3535 
0.5000 



220 and the feature extractmg means 2j 
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E(2) = 



0.7500 


-0.2500 


0.7500 


-0.2500 


0.7500 


-0.2500 
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0.3964 
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f0094H 0944 In the basis vector calculating means 220, initialization is performed with 
setting the basis vector W2(l) at [0.0100, -0.0100, 0.0100, -0.0100, 0.0100, -0.0100, 
0.0100, -0.0100, 0.0100, -0.0100, 0.0100]', at a fixed value 0.1, i3 2 at 1 x lO"^. 
The calculation shown in Fig. 1 1 is performed for a hundred nineteen times. Then, the 
basis vector W2 = [0.3162, 0.3162, 0.3162, 0.3162, 0.3162, -0.3162, -0.3162, -0.3162, 
-0.3162, -0.3162, 0.0000]' is stored in the basis vector data file 300, and output to the 
feature extracting means 230 and the term-document matrix updating means 210. 
[00951f 095| * First Repetition in Basis Vector Calculating Means 220 (k = 1) 
[00961f O96| From the foregoing expression (8), 

W2(2) - [0.0102, -0.0098, 0.0102, -0.0096, 0.0104, -0.0105, 0.0095, -0.0103, 0.0097, 
-0.0105, 0.0102]' 

W2(2) - W2(l) = 10"^ X [0.2154, 0.2156, 0.2154, 0.3822, 0.3821, -0.4511, -0.4513, 
-0.2844, -0.2846, -0.451 1, 0.1666]' 
6 2(1) = 0.0011 

[00971f 09^ * Second Repetition in Basis Vvector Calculating Means 220 (k = 2) 
100981fO98} From the foregoing expression (8), 

W2(3) = [0.0105, -0.0095, 0.0105, -0.0092, 0.0108, -0.0110, 0.0090, -0.0106, 0.0094, 
-0.0110, 0.0103]' 
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W2(3) - W2(2) = 10-^ X [0.2624, 0.2626, 0.2624, 0.4413, 0.4411, -0.5152, -0.5154, 
-0.3364, -0.3366, -0.5152, 0.1786]' 

6 2(2) = 0.0013 
I00991f099} ~ syncopated ~ 

[0100] * A Hundred Nineteenth Repetition in Basis Vector Calculating Means 220 
(k= 119) 

fOlOll From the foregoing expression (8), 

W2(120) = [0.3162, 0.3162, 0.3162, 0.3162, 0.3162, -0.3162, -0.3162, -0.3162, -0.3162, 
0.0000]' 

W2(120) - W2(119) = 10"^ X [0.3327, 0.3333, 0.3327, -0.1375, -0.1381, 0.3332, 0.3326, 
-0.1377, -0.1383, 0.3332, -0.4712]' 

6 2(1 19) = 9.8141 X 10 ' 
[0102] In the feature extracting means 230, the operations shown in the expressions 
(11) and (12) are performed for outputting: 
yz =[0.7071, -0.7071, -0.0000] 

and 

P2 = 2.2361 

to the feature data file 400 and the normalizing parameter data file 450. 
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[01031 From the results set forth above, the feature vectors of the documents 1, 2 
and 3 in Fig. 1 are respectively [0.5000, 0.7071]', [0.5000, -0.7071], [0.7071, -0.0000]. 
Comparing these with the features of the LSA of respective documents shown in the 
explanation of the prior art, the second element of each vector is of opposite sign but 
has the same absolute value. Accordingly, concerning calculation of similarity in the 
expression (2), they have the same nature as the features of LSA. 
[01041 B. Question of Fig. 3 

[01051 Here, let us use the basis vectors stored in the basis vector data file 300 and 
the normalizing parameters stored in the normalizing parameter data file 450 during 
extraction of the features of the documents of Fig. 1 . Thereby, execution of the basis 
vector calculating means 220 and calculation of the normalizing parameter in the 
feature extracting means 230 are omitted. Let X denote the term-document matrix of 
Fig. 4. 

[01061 L First Iteration in Feature Extracting Means 200 (i = 1) 
^ ^ s: I S [01071 In the term-document matrix updating means 210, E(l) expressed as follows 

9 "1 o S r 

01 ^ J E- S 5 

W * H M H 2 

Q < 5 ^ ^ ^ fro"^ the foregoing expression (5) is output to the feature extracting means 230. 

^ 5 5 S t« u 
5 w F u < 
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0 
0 



E(l) = 



0 



0 



[01081 In the feature extracting means 230, arithmetic operation according to the 
foregoing expressions (11) and (12) is performed using the feature vector Wi and the 
normalizing parameter p\ obtained upon extraction of the features of the documents of 
Fig. 1 to output 

yi - [0.6542] 

to the feature data file 400. 
[01091 II. Second Iteration in Feature Extraction Control Means 200 (i = 2) 
[01101 In the term-document matrix updating means 210, using the feature vector 




Wi obtained upon performing feature extraction of the documents shown in Fig. 1, from 



the foregoing equation (5), E(2) expressed as follows is output to the feature extracting 



means 230. 
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E(2) = 



-0.3271 
-0.3271 
0.6729 
0.2103 
0.2103 
0.6729 
0.6729 
-0.7897 
0.2103 
-0.3271 
-0.4626 



[01111 In the feature extracting means 230, arithmetic operation according to the 
foregoing expressions (11) and (12) is performed using the feature vector W2 and the 
normalizing parameter pi obtained upon extraction of the features of the documents of 



Fig. 1 to output 



y2 = [-0.0000] 



to the feature data file 400. 



[01121 From the result set forth above, the feature vector of the question of Fig. 3 
becomes [0.6542, -0.0000]', comparing the value explained in the prior art, the second 



element has the same absolute value. 



101131 The present invention has been described in detail with respect to preferred 
embodiments. It will now be apparent fi-om the foregoing to those skilled in the art that 



changes and modifications may be made without departing from the invention in its 
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broader aspect. It is the intention, therefore, in the apparent claims to cover all such 
changes and modifications as fall within the true spirit of the invention. 
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ABSTRACT OF THE DISCLOSURE 
Concerning feature extraction of documents in text mining, a method and an 
apparatus for extracting features having the same nature as those by LSA are provided 
that require smaller memory space and simpler program and apparatus than the 
apparatus for executing LSA. Features of each document are extracted by feature 
extracting acts on the basis of a term-document matrix updated by term-document 
updating acts and of a basis vector, spanning a space of effective features, calculated by 
basis vector calculations. Execution of respective acts is repeated until a 
predetermined requirement given by a user is satisfied. 
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